{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6a32949c",
   "metadata": {},
   "source": [
    "![](https://europe-west1-atp-views-tracker.cloudfunctions.net/working-analytics?notebook=tutorials--fine-tunning-agents--fine-tuning-agents-guide)\n",
    "\n",
    "# Enhancing AI Agents Through Fine-Tuning Guide\n",
    "## Overview\n",
    "This tutorial demonstrates how to leverage supervised fine-tuning (SFT) to create more capable and efficient AI agents. While we'll use a banking example for demonstration, the techniques and principles covered here apply to any scenario where you need to enhance an AI agent's capabilities through fine-tuning.\n",
    "\n",
    "## What is Fine-Tuning?\n",
    "\n",
    "Fine-tuning is the process of taking a pre-trained AI model—one that has already learned general language patterns from vast amounts of data—and further training it on a smaller, specialized dataset. \n",
    "\n",
    "This additional training helps the model adapt to specific domains, tasks, or communication styles, making it more effective and reliable for targeted applications. Fine-tuning allows you to customize the model’s behavior, vocabulary, and output format to better suit your unique needs.\n",
    "\n",
    "## Why Fine-Tune AI Agents?\n",
    "Fine-tuning offers several powerful advantages for AI agent development:\n",
    "\n",
    "1. **Domain Expertise**: \n",
    "   - Internalizes domain-specific regulations and compliance requirements\n",
    "\n",
    "   - Reduces hallucinations in specialized contexts\n",
    "\n",
    "2. **Consistent Style & Tone**:\n",
    "   - Learns to maintain your brand voice consistently\n",
    "\n",
    "   - Provides uniform responses across all interactions\n",
    "\n",
    "3. **Structured Outputs**:\n",
    "   - Reliably generates specific formats (JSON, SQL, Markdown)\n",
    "\n",
    "4. **Reduced Prompt Engineering**:\n",
    "   - Eliminates need for lengthy system prompts\n",
    "\n",
    "   - Significantly reduces token usage and costs\n",
    "\n",
    "5. **Enhanced Efficiency**:\n",
    "   - Smaller tuned models can replace larger base models\n",
    "\n",
    "   - Faster response times for specialized tasks\n",
    "\n",
    "\n",
    "### Who Should Consider Fine-Tuning?\n",
    "- Teams building specialized AI agents for specific domains\n",
    "\n",
    "- Applications where response speed and cost matter\n",
    "\n",
    "- Systems handling complex domain-specific tasks\n",
    "\n",
    "## Method Overview\n",
    "1. **Data Preparation**: Creating high-quality training datasets that capture desired behaviors\n",
    "\n",
    "2. **Supervised Fine-Tuning**: Adapting foundation models to specific requirements\n",
    "\n",
    "3. **Integration**: Building efficient agent workflows with fine-tuned models\n",
    "\n",
    "4. **Evaluation**: Measuring performance improvements across key metrics\n",
    "\n",
    "\n",
    "This approach provides a robust framework for enhancing AI agents through fine-tuning, enabling them to perform specialized tasks more effectively while maintaining efficiency and reliability.\n",
    "\n",
    "## Conclusion\n",
    "This tutorial shows how to leverage fine-tuning to create more capable AI agents. Whether you need domain expertise, consistent formatting, or improved efficiency, fine-tuning can help your agents perform better while reducing costs and complexity. The techniques demonstrated here can be applied across any industry or use case where specialized agent behavior is desired.\n",
    "\n",
    "Written by [Zohar Mosseri](https://www.linkedin.com/in/zohar-mosseri/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d69c2ef0",
   "metadata": {},
   "source": [
    "📖 **For more background on fine-tuning AI models and their applications, check out our detailed blog post:** [Fine-tuning AI models](https://diamantai.substack.com/p/fine-tuning-ai-models-how-they-evolve)\n",
    "\n",
    "*DiamantAI is a top 0.1% newsletter for AI with over 25,000 subscribers, focusing on AI techniques, breakthroughs, and tutorials.*\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32505f5d",
   "metadata": {},
   "source": [
    "## Example Use Case: Banking Assistant\n",
    "To demonstrate these concepts, we'll create a banking customer support assistant. This example will show how to:\n",
    "- Structure training data for domain-specific knowledge\n",
    "- Fine-tune a model for specialized tasks\n",
    "- Integrate the fine-tuned model into a practical application"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de329bb2",
   "metadata": {},
   "source": [
    "## 1. Setup and Imports\n",
    "\n",
    "Install and import necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3c40d0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade openai langchain langgraph==0.3.1 tiktoken pandas scikit-learn\n",
    "\n",
    "import os, json, time\n",
    "from openai import OpenAI\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "# Load environment variables and set OpenAI API key\n",
    "load_dotenv()\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_API_KEY')\n",
    "\n",
    "# Initialize OpenAI client\n",
    "client = OpenAI()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c03509b2",
   "metadata": {},
   "source": [
    "## 2. Prepare Training & Validation Files for Fine-tuning\n",
    "\n",
    "OpenAI expects **UTF‑8 encoded JSONL** files for fine-tuning, with one JSON object per line.\n",
    "Each example should follow the chat format with system, user, and assistant messages.\n",
    "\n",
    "### Understanding the Training Data Format\n",
    "The quality of your fine-tuned model depends heavily on your training data. Here's what you need to know:\n",
    "\n",
    "1. **File Format**: JSONL (JSON Lines) with UTF-8 encoding\n",
    "\n",
    "2. **Message Structure**: Each example must contain a sequence of messages with roles and content\n",
    "\n",
    "3. **Required Fields**: Each message needs `role` and `content` fields\n",
    "\n",
    "4. **Valid Roles**: `system`, `user`, and `assistant`\n",
    "\n",
    "### Example Format\n",
    "```jsonl\n",
    "{\n",
    "  \"messages\": [\n",
    "    {\"role\": \"system\", \"content\": \"You are a helpful domain expert assistant.\"},\n",
    "    {\"role\": \"user\", \"content\": \"What is the policy on this matter?\"},\n",
    "    {\"role\": \"assistant\", \"content\": \"According to our policies...\"}\n",
    "  ]\n",
    "}\n",
    "```\n",
    "\n",
    "### Creating Training Data: A Banking Example\n",
    "To demonstrate these principles, let's create a training dataset using a banking knowledge base. This example can be adapted for any domain-specific knowledge:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13333b0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import random\n",
    "import pathlib\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "# Create directory for our fine-tuning data\n",
    "DATA_DIR = pathlib.Path(\"bank_finetune_data\")\n",
    "DATA_DIR.mkdir(exist_ok=True)\n",
    "\n",
    "# --- General Approach to Creating Training Data ---\n",
    "# 1. Use your domain knowledge base (docs, FAQs, policies, etc.)\n",
    "# 2. Structure content into clear topics\n",
    "# 3. Create diverse, well-phrased Q&A pairs\n",
    "\n",
    "# Example using banking domain\n",
    "kb_docs = [\n",
    "    {\n",
    "        \"title\": \"Account Types\",\n",
    "        \"content\": {\n",
    "            \"checking\": \"We offer three types of checking accounts: Basic (no minimum balance), Premium ($2,500 minimum, no fees), and Student (no fees with valid student ID).\",\n",
    "            \"savings\": \"Our savings accounts include Regular (0.5% APY), High-Yield (1.5% APY with $10,000 minimum), and Goal-Based savings with customizable targets.\",\n",
    "            \"business\": \"Business accounts feature unlimited transactions, merchant services integration, and dedicated support. Available in Standard and Premium tiers.\"\n",
    "        }\n",
    "    },\n",
    "    # Additional examples below - uncomment to create a more comprehensive training set\n",
    "    # For tutorial purposes, we'll start with a single example to demonstrate the process\n",
    "    # {\n",
    "    #     \"title\": \"Security Protocols\",\n",
    "    #     \"content\": {\n",
    "    #         \"authentication\": \"We use multi-factor authentication, biometric verification, and device recognition. Login attempts are limited to 3 tries before temporary lockout.\",\n",
    "    #         \"fraud_prevention\": \"Real-time transaction monitoring, instant fraud alerts, and zero liability protection for unauthorized charges.\",\n",
    "    #         \"data_protection\": \"End-to-end encryption for all transactions, secure socket layer (SSL) protection, and regular security audits.\"\n",
    "    #     }\n",
    "    # },\n",
    "    # {\n",
    "    #     \"title\": \"Digital Banking\",\n",
    "    #     \"content\": {\n",
    "    #         \"mobile_features\": \"Mobile check deposit, peer-to-peer payments, bill pay, and account aggregation across institutions.\",\n",
    "    #         \"online_services\": \"Budget tracking, spending analysis, electronic statements, and customizable alerts.\",\n",
    "    #         \"technical_requirements\": \"Compatible with iOS 13+ and Android 8+. Requires secure internet connection and updated browser versions.\"\n",
    "    #     }\n",
    "    # }\n",
    "]\n",
    "\n",
    "def generate_diverse_examples(kb_doc):\n",
    "    \"\"\"\n",
    "    Generate multiple training examples with diverse phrasings and scenarios.\n",
    "    Following best practices for high-quality training data.\n",
    "    \"\"\"\n",
    "    examples = []\n",
    "    content = kb_doc[\"content\"]\n",
    "    \n",
    "    # Generate examples for each subtopic\n",
    "    for subtopic, details in content.items():\n",
    "        # Multiple question variations\n",
    "        questions = [\n",
    "            f\"Can you explain {subtopic}?\",\n",
    "            f\"What should I know about {subtopic}?\",\n",
    "            f\"Tell me about your {subtopic}\",\n",
    "            f\"How does {subtopic} work?\",\n",
    "            f\"I need information regarding {subtopic}\",\n",
    "            f\"What are the details of {subtopic}?\"\n",
    "        ]\n",
    "        \n",
    "        # Multiple response variations with consistent style\n",
    "        responses = [\n",
    "            f\"Here's what you need to know about {subtopic}: {details}\",\n",
    "            f\"Regarding {subtopic}: {details} Let me know if you need any clarification.\",\n",
    "            f\"I'll explain {subtopic}. {details} Is there anything specific you'd like to know more about?\",\n",
    "            f\"{details} This information about {subtopic} is current as of today. Please check our website for any updates.\"\n",
    "        ]\n",
    "        \n",
    "        # Generate combinations with system message variations\n",
    "        system_messages = [\n",
    "            \"You are a knowledgeable banking assistant focused on providing accurate, compliant information.\",\n",
    "            \"You are a helpful financial services expert committed to clear, precise communication.\",\n",
    "            \"You are a banking specialist dedicated to providing detailed, accurate responses.\"\n",
    "        ]\n",
    "        \n",
    "        # Create multiple examples for each subtopic\n",
    "        for system_msg in system_messages:\n",
    "            for question in questions:\n",
    "                for response in responses:\n",
    "                    examples.append({\n",
    "                        \"messages\": [\n",
    "                            {\"role\": \"system\", \"content\": system_msg},\n",
    "                            {\"role\": \"user\", \"content\": question},\n",
    "                            {\"role\": \"assistant\", \"content\": response}\n",
    "                        ]\n",
    "                    })\n",
    "    \n",
    "    return examples\n",
    "\n",
    "# Generate comprehensive training dataset\n",
    "examples = []\n",
    "for doc in kb_docs:\n",
    "    examples.extend(generate_diverse_examples(doc))\n",
    "\n",
    "# Shuffle examples to ensure good distribution\n",
    "random.shuffle(examples)\n",
    "\n",
    "print(f\"Generated {len(examples)} diverse training examples\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68521ee3",
   "metadata": {},
   "source": [
    "### Data Quality Best Practices for Fine-Tuning\n",
    "\n",
    "The success of your fine-tuned model depends on well-structured training data. Key principles:\n",
    "\n",
    "1. **Data Requirements**\n",
    "   - At least 10 examples per behavior\n",
    "   - Balance types and reserve 10–20% for validation\n",
    "\n",
    "2. **Quality Guidelines**\n",
    "   - Clear intent and response in each example\n",
    "   - Vary phrasing, keep formatting and style consistent\n",
    "   - Reflect real-world usage\n",
    "\n",
    "3. **Common Pitfalls**\n",
    "   - Avoid sensitive info, inconsistent styles, duplicates, and overfitting\n",
    "\n",
    "Remember: Quality over quantity—a small set of high-quality examples outperforms a large set of poor ones."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1742ea9f",
   "metadata": {},
   "source": [
    "### Split into training and validation sets\n",
    "\n",
    "A crucial step in fine-tuning is dividing your dataset into training and validation sets. The training set teaches the model, while the validation set helps measure its performance on unseen data. We'll use a 80/20 split - a common ratio that provides enough validation data while maximizing training examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e156e6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Randomly shuffle examples to ensure even distribution between sets\n",
    "random.shuffle(examples)\n",
    "\n",
    "# Calculate split point for 80% training, 20% validation\n",
    "split_index = int(0.8 * len(examples))\n",
    "\n",
    "# Split examples into training and validation sets\n",
    "train_examples, validation_examples = examples[:split_index], examples[split_index:]\n",
    "\n",
    "# Save to JSONL files\n",
    "train_file = DATA_DIR / \"train.jsonl\"\n",
    "validation_file = DATA_DIR / \"validation.jsonl\"\n",
    "\n",
    "with open(train_file, \"w\", encoding=\"utf-8\") as f:\n",
    "    for example in train_examples:\n",
    "        f.write(json.dumps(example, ensure_ascii=False) + \"\\n\")\n",
    "\n",
    "with open(validation_file, \"w\", encoding=\"utf-8\") as f:\n",
    "    for example in validation_examples:\n",
    "        f.write(json.dumps(example, ensure_ascii=False) + \"\\n\")\n",
    "\n",
    "print(f\"✍️ Created {len(train_examples)} training examples and {len(validation_examples)} validation examples\")\n",
    "\n",
    "# Create a DataFrame for easy visualization of example structure\n",
    "df = pd.DataFrame([\n",
    "    {\n",
    "        \"system\": ex[\"messages\"][0][\"content\"],\n",
    "        \"user\": ex[\"messages\"][1][\"content\"],\n",
    "        \"assistant\": ex[\"messages\"][2][\"content\"]\n",
    "    }\n",
    "    for ex in train_examples[:3]\n",
    "])\n",
    "# Display first 3 examples showing system, user, and assistant messages\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d068ba05",
   "metadata": {},
   "source": [
    "### Validating the JSONL format\n",
    "\n",
    "Before uploading, let's validate our files to ensure they meet OpenAI's requirements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20230de6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate_jsonl(file_path):\n",
    "    \"\"\"Validate that the file is proper JSONL format for fine-tuning.\"\"\"\n",
    "    with open(file_path, 'r', encoding='utf-8') as f:\n",
    "        line_count = 0\n",
    "        for line in f:\n",
    "            line_count += 1\n",
    "            try:\n",
    "                data = json.loads(line)\n",
    "                # Check if it has the messages field\n",
    "                if 'messages' not in data:\n",
    "                    return False, f\"Line {line_count} missing 'messages' field\"\n",
    "                # Check message format\n",
    "                for msg in data['messages']:\n",
    "                    if 'role' not in msg or 'content' not in msg:\n",
    "                        return False, f\"Line {line_count} has message missing 'role' or 'content'\"\n",
    "                    if msg['role'] not in ['system', 'user', 'assistant']:\n",
    "                        return False, f\"Line {line_count} has invalid role: {msg['role']}\"\n",
    "            except json.JSONDecodeError:\n",
    "                return False, f\"Line {line_count} is not valid JSON\"\n",
    "    return True, f\"Validated {line_count} examples\"\n",
    "\n",
    "# Validate our files\n",
    "train_valid, train_msg = validate_jsonl(train_file)\n",
    "val_valid, val_msg = validate_jsonl(validation_file)\n",
    "\n",
    "print(f\"Training file: {'✅' if train_valid else '❌'} {train_msg}\")\n",
    "print(f\"Validation file: {'✅' if val_valid else '❌'} {val_msg}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "affb7a67",
   "metadata": {},
   "source": [
    "## 3. Upload your JSONL files to OpenAI and Launch Fine-tuning\n",
    "\n",
    "We'll upload our files programmatically using the OpenAI Python client:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7947c0cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "# Initialize OpenAI client\n",
    "client = OpenAI(api_key=os.environ[\"OPENAI_API_KEY\"])\n",
    "\n",
    "def upload(path):\n",
    "    \"\"\"Upload a file to OpenAI for fine-tuning\"\"\"\n",
    "    with open(path, \"rb\") as file:\n",
    "        resp = client.files.create(\n",
    "            file=file,\n",
    "            purpose=\"fine-tune\"  # Specify the file will be used for fine-tuning\n",
    "        )\n",
    "    print(f\"Uploaded {Path(path).name} → {resp.id}\")\n",
    "    return resp.id\n",
    "\n",
    "# Upload our training and validation files\n",
    "# These IDs will be used when creating the fine-tuning job\n",
    "train_file_id = upload(train_file)\n",
    "validation_file_id = upload(validation_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a62d24e",
   "metadata": {},
   "source": [
    "You can also upload them through the UI: \n",
    "\n",
    "1. Navigate to the dashboard > [fine-tuning](https://platform.openai.com/finetune).\n",
    "\n",
    "2. Click + Create.\n",
    "\n",
    "3. Under Training data, upload your JSONL files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9755c462",
   "metadata": {},
   "source": [
    "## 4.  Launch the fine‑tuning job\n",
    "### Create fine-tuning job with the base model and the JSONL files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "890d4ab6",
   "metadata": {},
   "outputs": [],
   "source": [
    "job = client.fine_tuning.jobs.create(\n",
    "    # The ID of the JSONL file containing training examples\n",
    "    training_file=train_file_id,\n",
    "    # The ID of the JSONL file containing validation examples (optional but recommended)\n",
    "    validation_file=validation_file_id,\n",
    "    model=\"gpt-4o-mini-2024-07-18\",  # base model\n",
    "    suffix=\"banking-support\"  # custom suffix for the fine-tuned model\n",
    ")\n",
    "# Print job information for monitoring\n",
    "print(f\"Fine-tuning job created with ID: {job.id}\")\n",
    "print(f\"Job status: {job.status}\")\n",
    "print(f\"Monitor progress at: https://platform.openai.com/finetune\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c69541e",
   "metadata": {},
   "source": [
    "## 5.  Monitor job status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b866c983",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "# Continuously check job status every 30 seconds\n",
    "while True:\n",
    "    # Retrieve current status of the fine-tuning job\n",
    "    status = client.fine_tuning.jobs.retrieve(job.id).status\n",
    "    print(\"Status:\", status)\n",
    "    if status in (\"succeeded\", \"failed\", \"cancelled\"):\n",
    "        break      # Exit loop if job reaches a terminal state\n",
    "    # Wait 30 seconds before next status check\n",
    "    time.sleep(30)\n",
    "\n",
    "# Once complete, get the identifier for the fine-tuned model\n",
    "fine_tuned_model = openai.fine_tuning.jobs.retrieve(job.id).fine_tuned_model  # e.g. 'ft:gpt‑4o-mini:banking-support:2025-05-12-10-30-02'\n",
    "print(\"✅ Fine‑tuned model:\", fine_tuned_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6337528",
   "metadata": {},
   "source": [
    "### Understanding Fine-Tuning Metrics\n",
    "\n",
    "You can watch progress and loss curves in the [fine-tuning](https://platform.openai.com/finetune) tab of the OpenAI dashboard. The fine-tuning process typically takes a few minutes to several hours depending on dataset size and model complexity.\n",
    "\n",
    "\n",
    "When fine-tuning, you'll see two key graphs:\n",
    "\n",
    "#### Loss Curves\n",
    "The loss curves show how well the model is learning over time:\n",
    "\n",
    "- **Training Loss** (🟢 Green): Model's error on training data (should decrease)\n",
    "\n",
    "- **Validation Loss** (🟣 Purple): Error on unseen data (should follow training loss)\n",
    "\n",
    "- If validation loss increases while training loss decreases = overfitting\n",
    "\n",
    "#### Accuracy Curves\n",
    "These show the model's prediction accuracy over time:\n",
    "\n",
    "- **Training Accuracy** (🟢 Green): Should steadily increase\n",
    "\n",
    "- **Validation Accuracy** (🟣 Purple): Should follow training curve\n",
    "- If they diverge = potential overfitting\n",
    "\n",
    "#### What to Look For\n",
    "✅ **Good Signs**\n",
    "- Both metrics improving steadily\n",
    "\n",
    "- Training and validation curves following similar trends\n",
    "\n",
    "⚠️ **Warning Signs**\n",
    "- Validation loss increasing while training loss decreases\n",
    "\n",
    "- Large gaps between training and validation metrics\n",
    "\n",
    "- Erratic or unstable curves\n",
    "\n",
    "![Fine-tuning metrics showing loss and accuracy curves](https://i.ibb.co/hx55yKKH/loss-accuracy-curve.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a6d1e71",
   "metadata": {},
   "source": [
    "## 6. Implementing Your Fine-Tuned Model\n",
    "\n",
    "This section demonstrates how to integrate your fine-tuned model into a production environment using LangGraph. You'll learn how to:\n",
    "- Connect your specialized model to existing tools and systems\n",
    "\n",
    "- Structure the interaction flow\n",
    "\n",
    "- Handle domain-specific queries effectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5420c16c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.tools import tool\n",
    "from langchain_openai import ChatOpenAI\n",
    "from langchain_core.messages import HumanMessage\n",
    "from langgraph.prebuilt import create_react_agent\n",
    "\n",
    "\n",
    "# Define example tool to demonstrate fine-tuned model integration\n",
    "@tool\n",
    "def account_lookup(account_id: str) -> str:\n",
    "    \"\"\"Look up account information by account ID (requires authentication).\"\"\"\n",
    "    return f\"Account {account_id} information is available after authentication.\"\n",
    "\n",
    "@tool\n",
    "def transfer_payment(source_account: str, destination_account: str, amount: float) -> str:\n",
    "    \"\"\"Transfer payment from one account to another.\"\"\"\n",
    "    return f\"Payment of {amount} from {source_account} to {destination_account} has been initiated.\"\n",
    "\n",
    "\n",
    "# Use our fine-tuned model\n",
    "domain_expert = ChatOpenAI(model=fine_tuned_model, temperature=0.0)\n",
    "\n",
    "# Create the agent with our tools\n",
    "agent = create_react_agent(\n",
    "    model=domain_expert,  # Use our fine-tuned model\n",
    "    tools=[account_lookup, transfer_payment]\n",
    "    )\n",
    "\n",
    "# Test the agent with some example queries\n",
    "test_queries = [\n",
    "    \"What account types do you offer?\",\n",
    "    \"What are the fees for wire transfers?\",\n",
    "    \"I need information about overdraft protection\"\n",
    "]\n",
    "\n",
    "# Run the agent on a test query\n",
    "for query in test_queries:\n",
    "    response = agent.invoke({\"messages\": [HumanMessage(content=query)]})\n",
    "    print(\"\\nQuery:\", query)\n",
    "    print(\"\\nResponse:\")\n",
    "    # Extract and print only the final assistant response\n",
    "    for msg in reversed(response[\"messages\"]):\n",
    "        # Check if it's an AIMessage and has non-empty content\n",
    "        if hasattr(msg, \"content\") and msg.content and msg.__class__.__name__ == \"AIMessage\":\n",
    "            print(msg.content)\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d35ffb2",
   "metadata": {},
   "source": [
    "*Tip: if you later need multiple specialised models, simply point each LangGraph node to a different fine tuned model.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4381050",
   "metadata": {},
   "source": [
    "## 7. Benefits of Fine-Tuning: A Comparative Analysis\n",
    "\n",
    "Let's compare how a fine-tuned model performs against a base LLM in two key areas:\n",
    "\n",
    "### 1. Domain Expertise & Security\n",
    "**Query:** \"I want to transfer $50,000 to account 1234567\"\n",
    "\n",
    "**Base LLM:**\n",
    "> For your safety, please make sure you are using a secure and verified banking platform to perform this action. If you’d like, I can draft a sample confirmation message or explain the common steps to transfer funds through your bank or app. Would you like me to do that?\n",
    "\n",
    "**Fine-Tuned Model:**\n",
    "> For your security, I'll need to:\n",
    "> 1. Verify your identity through our authentication process\n",
    "> 2. Confirm the destination account details\n",
    "> 3. Check if this transfer requires additional fraud prevention measures (required for transfers over $10,000)\n",
    "> \n",
    "> Please provide your account ID to begin the secure verification process.\n",
    "\n",
    "### 2. Structured Responses & Product Knowledge\n",
    "**Query:** \"What savings accounts do you offer?\"\n",
    "\n",
    "**Base LLM:**\n",
    "> Banks typically offer various types of savings accounts with different interest rates and minimum balance requirements. You should check with your specific bank for their offerings.\n",
    "\n",
    "**Fine-Tuned Model:**\n",
    "```json\n",
    "{\n",
    "    \"savings_products\": {\n",
    "        \"regular_savings\": {\n",
    "            \"rate\": \"0.5% APY\",\n",
    "            \"min_balance\": \"$0\",\n",
    "            \"features\": [\"Mobile banking\", \"24/7 support\"]\n",
    "        },\n",
    "        \"high_yield_savings\": {\n",
    "            \"rate\": \"1.5% APY\",\n",
    "            \"min_balance\": \"$10,000\",\n",
    "            \"features\": [\"Premium interest rate\", \"Priority support\"]\n",
    "        }\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "**Takeaway:**  \n",
    "\n",
    "These improvements would be difficult to achieve through prompt engineering alone, showing the value of fine-tuning for creating specialized AI agents."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c18598c",
   "metadata": {},
   "source": [
    "## 8.  Best Practices for Fine-Tuning\n",
    "\n",
    "| Topic                   | Recommendation                                                                                   |\n",
    "|-------------------------|--------------------------------------------------------------------------------------------------|\n",
    "| **Train/Validation Split** | Always reserve at least 10% of your data (ideally ≥40 examples) for validation. This helps you monitor overfitting and ensures your model generalizes well to new queries. |\n",
    "| **Early Stopping**         | Enabled by default. Training will automatically stop if the validation loss stops improving, preventing overfitting and saving compute costs. |\n",
    "| **Example Truncation**     | Any sample longer than the model’s context window is auto‑truncated. Keep messages concise and relevant for best results. |\n",
    "| **Evaluation Metrics**     | The OpenAI dashboard shows token‑level accuracy. For deeper analysis, consider running BLEU or exact-match metrics on your own test set. |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff573867",
   "metadata": {},
   "source": [
    "## 9. Next Steps\n",
    "\n",
    "- Automate Data Refresh: \n",
    "  Schedule regular (e.g., nightly) re-training with new domain data to keep the model up-to-date with evolving knowledge and requirements.\n",
    "\n",
    "- Add Compliance & Audit Nodes:\n",
    "  Integrate additional LangGraph nodes (potentially with their own fine-tuned models) to automatically check responses for compliance and flag sensitive topics.\n",
    "\n",
    "- Monitor agent performance and costs with tools like LangSmith, and continuously evaluate real user queries to iterate and improve your training data and workflow."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
